Self-adjusting music scrolling system

ABSTRACT

A system for automatically adjusting a music score comprises a display screen and a music scrolling system configured to perform the steps of selecting a music score, the music score being cataloged and stored in a music score database as a first MIDI file, displaying a first portion of the selected music score on the display screen, recording musical notes with a digital device, storing the recorded musical notes in memory as a WAV file, converting the WAV file into a second MIDI file, comparing the first MIDI file and the second MIDI file with a MIDI comparison algorithm, determining if the first MIDI file substantially matches the second MIDI file, automatically adjusting the music score on the display screen upon determining that the first MIDI file substantially matches the second MIDI file, and displaying one or more mistakes upon determining that the first MIDI file does not substantially match the second MIDI file.

CROSS-REFERENCE TO RELATED APPLICATION(S)

The present application is a continuation application of applicationSer. No. 12/100,125 filed Apr. 9, 2008, entitled “Self-Adjusting MusicScrolling System,” which is hereby incorporated by reference in itsentirety.

FIELD OF THE INVENTION

The present invention relates in general to the display of words ormusic notes, and more particularly, to an automatic scrolling system andmethod of adjusting a music score shown on a display based uponrecognition of the music notes or words already played or sung by amusician.

BACKGROUND OF THE INVENTION

Previous systems and methods have attempted to provide electronic musicdisplays as an alternative to requiring musicians to manually turnpages. However, such solutions have not conquered the many obstaclesthat exist in providing electronic music displays because, for example,they require manual intervention, either directly to activate thescrolling or as a consequence of mistakes made by the musician (i.e.,playing the wrong note).

More particularly, existing solutions are generally not automatic and/orare not synchronized with the execution of music notes played by themusician. These solutions exempt the musician from undertaking manualactions in only very limited circumstances. For example, in one existingsystem designed for use by an orchestra, a cursor is displayed on ascreen containing sheet music in order to indicate the current beat setby the director. However, if the director makes a mistake, the musicianis then forced to determine for himself what the correct position of thecursor should be. In another existing system, the musician may activatethe scrolling of music on a display by using his or her foot. However,for obvious reasons, this type of system may not be used in conjunctionwith instruments that are played, at least in part, with the feet (e.g.a piano or an organ). In yet another existing system, automatictime-based scrolling is used to scroll the music notes on a displaybased upon a predetermined time interval. However, the automaticscrolling solution utilizing a customizable time interval is often timesinaccurate, thereby scrolling either ahead of or behind time anddisrupting the musician's ability to play the proper notes at the propertempo.

Furthermore, existing solutions do not adequately take into account thefact that reading “sheet” music is not always sequential like reading abook. For example, when reading sheet music, sets of beats may berepeated and may terminate differently at any repetition. Furthermore,under certain circumstances, there may be “jumps” from one page toanother page.

Based on the foregoing, there exists a need for an improved system andmethod for automatically scrolling music on a display based uponrecognition of the music notes previously played or sung by a musician.

BRIEF SUMMARY OF THE INVENTION

The present invention solves the foregoing problems by providing asystem for automatically adjusting a music score comprising a displayscreen and a self-adjusting music scrolling system having a music scoredatabase, a memory, and a digital device for recording musical notes,the music scrolling system configured to perform the steps of selectinga music score to be played by a musician, wherein the music score iscataloged and stored in the music score database as a first MIDI file,displaying a first portion of the selected music score on the displayscreen, recording musical notes with the digital device, storing therecorded musical notes in the memory as a WAV file, converting the WAVfile into a second MIDI file, comparing the first MIDI file and thesecond MIDI file with a MIDI comparison algorithm, determining if thefirst MIDI file substantially matches the second MIDI file,automatically adjusting the music score on the display screen to show asecond portion of the selected music score upon determining that thefirst MIDI file substantially matches the second MIDI file, anddisplaying one or more mistakes detected on the display screen upondetermining that the first MIDI file does not substantially match thesecond MIDI file. The musical notes played by the musician may be playedon any type of instrument, including both acoustic and electronicinstruments. The comparison algorithm compares both pitch and durationparameters of musical notes in order to determine if the first MIDI filesubstantially matches the second MIDI file.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flowchart illustrating one embodiment of an automatic musicscrolling method in accordance with the present invention.

FIG. 2 is a flowchart illustrating an alternative embodiment of anautomatic scrolling system and method in accordance with the presentinvention that utilizes speech recognition.

DETAILED DESCRIPTION OF THE INVENTION

The present invention includes an automatic music scrolling system andmethod capable of adjusting a music score or notes shown on a displayscreen based upon recognition of the music notes already played or sungby a musician. In particular, one exemplary embodiment of the presentinvention is designed to both recognize the instrument timbre as well asthe music played. The invention generally combines music recognitionwith an electronic music display that visually depicts the real-timeposition in the music score. Thus, the automatic scrolling system andmethod recognizes the notes already played, tracks the performance onthe displayed music score, and advances the music score on the screenaccordingly and without the need for intervention by the musician.

In accordance with the present invention, the musician may be able toselect the type of scrolling suitable for him, such as by beats, linesor pages. Additionally, with regard to page scrolling the musician maybe able to select when the system scrolls to the next page. For example,the musician may select the number of beats remaining on a page beforethe system scrolls to the next page. With regard to line and beatscrolling, the musician may choose the fixed position of the currentline or beat. The musician may also select that a preview of the nextbeats/lines/pages be shown on a portion of the screen to help themusician visualize the notes that will be coming up shortly. It isimportant to note that it is not necessary for the automatic scrollingsystem of the present invention to always scroll sequentially. Forexample, the system may be configured such that it may automaticallyjump pages, in either direction, as required by the music score.

FIG. 1 is a flowchart illustrating one exemplary embodiment of a method10 for providing automatic scrolling of a previously selected musicscore. The method begins at step 12 where the musical notes played by amusician are recorded and stored in memory by a digital device in WAVformat. The recording of the musical notes played by the musician may beaccomplished using any one of a variety of commercially available WAVrecording devices, as will be appreciated by those skilled in the art.

The method continues at step 14 where the WAV file recorded and storedin step 12 is converted into a MIDI file. The conversion between a WAVand a MIDI file in step 14 may be performed by any one of a variety ofWAV to MIDI programs as will be appreciated by those skilled in the art.For example, one type of solution that may be used for the WAV to MIDIconversion of step 14 is a software program that uses an algorithmconfigured to take a sound signal as it is input and attempts to capturethe loudness, pitch, dynamics, and performance style present in thesound signal. All of these features captured by the program are thenwritten into a MIDI file. When replayed, this MIDI file will sound verysimilar to the original sound signals input into the converter. Anothersolution that may be used for the WAV to MIDI conversion of step 14includes a real-time WAV to MIDI converter that is capable of analyzinga live or recorded solo performance (such as a singing voice or amusical instrument) and converting the corresponding sound signals intoa standard MIDI file. This type of software program may not only becapable of reading standard WAV audio files, but also may accept livemicrophone input.

The method continues at step 16 where the MIDI file created in step 14is compared to a MIDI file corresponding to the music score previouslyselected by the musician, which may be cataloged in and accessed from amusic repository database. The music repository database preferablystores the music scores in MIDI format, although those skilled in theart will appreciate that the music scores may be stored in anotherformat, such as in WAV format, and later converted into a MIDI file suchas with the WAV to MIDI programs previously described. The comparisonbetween the MIDI file corresponding to the music notes recorded in step12 and the MIDI file corresponding to the predetermined music scorestored in the music repository database may be based on the eventsconstituting the MIDI format without the need to focus on all aspects ofthe event. For example, the comparison may be performed based on pitch(either relative or absolute) and/or the duration of the event. In oneembodiment, the comparison algorithm may place a greater weighting onthe duration of the music notes rather than on their pitch.

Next, in step 18, a determination is made whether or not the MIDI filecorresponding to the music notes recorded in step 12 substantially matchthe MIDI file corresponding to the pre-selected music score. Step 18preferably incorporates an “error allowance” that may be selected by themusician, thus providing the musician with the power to select thedegree of a “mistake” that is considered acceptable. In general, amistake may be one or more consecutive incorrect notes played by themusician. In case of a mistake within the allowance limit set by themusician, the automatic scrolling will continue tracking the performanceof the musician ignoring the incorrect notes. However, when the mistakeis above the allowance limit a mechanism of matching the next playedbeats with the music will take place and will re-synchronize themusician's performance to the music score. The musician may have theopportunity to select the search range of the re-synchronizationmechanism, for example in the same piece, a selected range of pieces,the same page, etc.

The definition of an error tolerance is needed because, after a mistake,the musician could stop and restart the performance from another point,either prior to or after the beat of the mistake. The possibility ofmodulating the allowance level will permit the musician to adjust thesynchronization mechanism to the ability of the musician in performingthe piece. The mechanism of matching played beats with music may alsoexempt the musician from having to select the piece before starting toplay. For example, the musician could simply start to play the firstbeats, which may thereafter be automatically synchronized to the rightpiece at the right beat.

If it is determined in step 18 that the MIDI file corresponding to themusic notes recorded in step 12 substantially match the MIDI filecorresponding to the pre-selected music score, then the method continuesat step 20 where the music score shown on the display is automaticallyscrolled so that the musician may continue playing without anyinterruption.

Alternatively, if it is determined in step 18 that the MIDI filecorresponding to the musical notes recorded in step 12 do notsubstantially match the MIDI file corresponding to the pre-selectedmusical score, then the method continues at step 22 where the user isprovided with feedback regarding the mistakes detected in the comparisonstep.

In a musical group, the synchronization in accordance with the presentinvention may also happen as a result of musical notes being played byother musicians. In particular, the music scrolling systems of the groupmay be connected and always synchronized, delegating the scrolling taskto only one instrument or by necessity, one scrolling system may betemporarily synchronized to another.

The tolerance of the music scrolling system in accordance with thepresent invention may largely depend on the tolerance of all theinvolved algorithms. For example, in the system and method previouslydescribed, the errors produced by the music recording and MIDIcomparison steps are usually negligible when compared to the errors thatmay be produced by the WAV to MIDI conversion step.

More particularly, the main source of error in the WAV to MIDI step isdue to the fact that several harmonics played by an instrument may beassociated with the same note. This type of error may greatly affect thetransposition of played music into music notation. However, it will notgenerally impact an automatic scrolling system in the same mannerbecause in such a system, the pitch of notes is not the only parameterthat may be relied upon. For example, another parameter that may be usedis duration. Thus, the scrolling system and method of the presentinvention may be set up based only on the pitch recognition, therebyintroducing a larger source of error. Alternatively, the scrollingsystem and method of the present invention may be set up based on theduration of the notes, thereby minimizing the potential errors. However,it may be desirable to use a combination of both of the methods in orderto reduce the potential errors by a considerable amount.

The system and method of the present invention has been described abovewith reference to musical notes generated by a musician playing aninstrument. However, the present invention may also be extended to wordsspoken or sung by an individual, which may then be compared to a script,music lyrics, or the like.

FIG. 2 is a flowchart illustrating one exemplary embodiment of a method100 for providing automatic scrolling based on speech recognition. Themethod operates similar to method 10 previously described, and thus,will be discussed only briefly. The method begins at step 102 where thevoice signals from words spoken or sung by an individual are recordedand stored in memory. The method continues at step 104 where the voicesignals recorded and stored in step 102 are converted into a text filewith speech recognition software. Then, in step 106, the text filecreated in step 104 is compared to a text file corresponding to a scriptor music lyrics previously selected by the individual. Next, in step108, a determination is made whether or not the text file correspondingto the words recorded in step 102 substantially matches the text filecorresponding to the pre-selected script or music lyrics. If it isdetermined in step 108 that the text file corresponding to the wordsrecorded in step 102 substantially matches the text file correspondingto the pre-selected script or music lyrics, then the method continues atstep 110 where the text shown on the display is automatically scrolledaccordingly. Alternatively, if it is determined in step 108 that thetext file corresponding to the words recorded in step 102 do notsubstantially match the text file corresponding to the pre-selectedscript or music lyrics, then the method continues at step 112 where theuser is provided with feedback regarding the mistakes detected in thecomparison step. As will be appreciated by those skilled in the art,method 100 may be adapted for use with numerous other speech-typeapplications without departing from the intended scope of the presentinvention.

One exemplary speech recognition method that may be used in conjunctionwith the present invention is the Hidden Markov Model (HMM). The HMMspeech recognition method is a statistical model which treats the voiceas a signal. Generally speaking, HMM receives a voice signal as an inputand produces an output that typically includes a representation (inplain text) of the recognized word or set of words.

Although the present invention has been described with reference topreferred embodiments, workers skilled in the art will recognize thatchanges may be made in form and detail without departing from the spiritand scope of the invention.

1. A system for automatically adjusting a music score shown on a displaycomprising: a display screen; and a self-adjusting music scrollingsystem having a music score database, a memory, and a digital device forrecording musical notes, the music scrolling system configured toperform the steps of: selecting a music score to be played by amusician, wherein the music score is cataloged and stored in the musicscore database as a first MIDI file; defining an acceptable musicianerror, wherein the acceptable musician error is a configurable parameterselected by the musician, and wherein the acceptable musician error maybe defined as the occurrence of one or more consecutive incorrectmusical notes played by the musician; displaying a first portion of theselected music score on the display screen; recording musical notesplayed by the musician with the digital device; storing the recordedmusical notes in the memory as a WAV file; converting the WAV file intoa second MIDI file; comparing the first MIDI file and the second MIDIfile with a MIDI comparison algorithm, wherein the comparison algorithmcompares both pitch and duration parameters of musical notes;determining if the first MIDI file substantially matches the second MIDIfile; executing an error monitoring sequence to determine if a mistakein the musical notes played by the musician exceeds the acceptablemusician error selected by the musician; automatically adjusting themusic score on the display screen to show a second portion of theselected music score upon determining that the first MIDI filesubstantially matches the second MIDI file; and re-synchronizing aposition of the musician within the selected music score upon detectinga mistake that exceeds the acceptable musician error defined by themusician, wherein re-synchronizing includes monitoring a string ofcorrect musical notes played after the mistake is detected and matchingthe string of correct musical notes with a corresponding string ofmusical notes in the selected music score.